Storage Stream Library
Storage Stream Library is a Delphi component for use in Win32 and Win64 (Windows XP/Vista/7/8/10), OSX, iOS, Android and Linux software and a COM class DLL for any developer environment that supports using of COM classes.
Reads and writes a versatile meta-data/binary stream format, for storing settings, presets or any kind of textual and/or binary data.
Storage Stream Library is basically a "create your own file format" component.
Features:
- Supports text and binary frames (with 11 built-in frame formats)
- Frames can be structured inside frames (multi-level hierarchy)
- Access directly all frame datas as a TMemoryStream (completely customized frame contents)
- Frames can be compressed and can be grouped individualy too
- Load the stream or dynamicaly access the stream frames on-demand from source medium
- Save the stream to a standard TStream, send the stream over network, store it in a database blob field, etc.
- 64 bit data sizes (supports files and content >4GB)
- Built-in encryption support (simple XOR or DES using OpenSSL)
- CRC32 integrity checking on stream and frame level
- Filters (queries) and sorting can be called on any branch level, like an SQL query
- Fully unicode implementation
- COM class DLL included
- Delphi XE2 64bit and OSX, Delphi XE5 iOS and Android, Lazarus/Free pascal compatible
A COM class DLL is also included that can be used for reading and writing Storage Stream Library files in any programming environment that supports using COM classes.
Storage Stream Library in shareware and commercial software?
The component comes with full source code (Delphi units) and can be evaluated freely. If you like it and decide to use it in a freeware, shareware or commercial (or any other money making - advertising, in app. selling, etc.) product one of the licenses is needed.
Requirements
Delphi 2009 and above.
To use the library with Lazarus/Free Pascal 'generics.collections.pas' is required (included in the 'Lazarus-FPC' folder, or download the latest version from: https://github.com/maciej-izak/generics.collections).
Installation
Add the directory to the search path, and to Uses list add: 'StorageStream'.
If you install the library with the setup.exe the COM classes will be registered automatically.
In other cases the COM classes need to be registered manually (don't forget to perform the regsvr32.exe in administrator mode!).
When you distribute your app. the COM classes must be registered too on the user's computer.
Inno setup example:
[Files]
Source: "D:\Storage Stream Library\Setup files\Bin\Win32\StorageStreamCOM.dll"; DestDir: "{app}\Bin\Win32"; Flags: regserver 32bit
Source: "D:\Storage Stream Library\Setup files\Bin\Win64\StorageStreamCOM.dll"; DestDir: "{app}\Bin\Win64"; Flags: regserver 64bit; Check: IsWin64
Manual registration: Run an elevated command prompt, change dir into the folder where 'StorageStreamCOM.dll' is located, run the following: "regsvr32 StorageStreamCOM.dll". This should register the COM class. Do this for the desired Win32 and/or Win64 version.
To use the library in VBScript, on a 64 bit OS, the VBScript process is 64 bit so the needed DLLs have to be from the Win64 folder, beside the .VBS file or in the System32 folder as written above.
Loading and opening
The Storage Stream can be accessed in 2 modes: either loading the whole stream into memory (LoadFromFile(), LoadFromStream()) or just scan the stream and parse the frame structure (OpenFile()/OpenStream()) and access the frame streams directly from the source medium (file/memory).
The latter can be used for "unlimited" size files as the frame stream data is not loaded into memory at all and only access a frame content on-demand.
If you opened the file stream in read and write mode it's not possible to modify the frame's content size but it's possible to overwrite it.
It's not possible to use the setter functions on a stream that was opened (not loaded), but you can call the frame's Deflash() method to load the frame content into memory and change it after if needed - the setter functions will work then.
Note: When opening a stream with OpenStream() the TStream must be alive until calling Close() or freeing the TStorageStream class.
Frame name encoding
By default frame names are written as unicode strings (UTF-16). Set 'FrameNameEncoding := ssfneUTF8;' to write UTF-8 frame IDs. UTF-8 format results smaller file sizes if no unicode characters are used for frame names, but requires a bit more processing so is a bit slower.
The library automatically reads the frame names according to the encoding specified in the stream.
There is a global 'StorageStreamGlobalDefaultFrameNameEncoding' variable that all newly created TSorageStream classes use automatically. 'TSorageStream.Clear' re-sets the encoding to the global default value.
Setting values
The root setter functions create new frame if one does not exists, or change the frame content if a frame specified by FrameID exists. Both functions work on frames belonging to the currently set 'GroupingContext' (see below 'Grouping').
To explicilty add a new text frame belonging to group 3 for example:
StorageStream.AddFrame('My new frame').SetText('Some text').GroupIdentifier := 3;
When using the SetStream() setter functions the 'DataStream' TStream object must be alive until calling 'TSorageStream.Clear' or freeing the 'TSorageStream' object.
Frames list
Use the provided AddFrame(), AddFrameFromFile(), AddFrameVirtual(), DeleteFrame(), DeleteAllFrames() and Clear() functions for manipulating the frames list. If you manipulate the Frames TList manually (eg. exchange items) always call ReIndexFrameList() afterwards so that to have the frames' Indexes valid.
FindFrame() and paths
The FindFrame() function supports asking for multiple levels with the '->' character pair. Do not use '->' in frame names!
Example frame structure:
Root
+Vehicles
+Ships
+Cars
+Ferrari
To get the 'Ferrari' frame write as:
MyFrame := StorageStream.FindFrame('Vehicles->Cars->Ferrari');
As the FindFrame() method is available at any level it can be used from any level to get the sub-frames.
The FrameExistsOrAdd(), Set*() (SetText() SetURL() etc.) and Get*() (GetText() GetURL() etc.), FilterBy*() and SortBy*() methods also support this.
Filters (queries) and sorting
Filters (queries) can be called on any branch level, works like an SQL query. The FilterBy*() and SortBy*() methods return interfaces so no need to free them and can be concatenated multiple times.
The frames are references to the global Storage Stream object's frames, only they are contained in a separate TList 'Frames', that can be accessed with .GetFrames.
Do not free the 'Frame' objects manually as they are managed by the 'StorageStream' object, use the frame's .Delete method instead when needed, then remove it from the filtered 'Frames' list if needed.
It's also possible to define the frame name as a path with the "->" character pair, eg.: 'Birth date->Sub frame->More sub-frame', please see above "FindFrame()" section.
This way it's possible to filter by any sub-sub-sub-frame.
StorageStreamFilter := Frame.FilterByDateRange('Person', 'Birth date', EncodeDate(1990, 1, 1), EncodeDate(2000, 1, 1)).FilterByIntegerRange('Person', 'Salary', 110000, 210000).SortByInteger('Salary');
The returned interfaces are actually TStorageStreamFramesBase instances.
To save the filtered list create a new 'TStorageStream' instance and use the 'AssignFramesByReference()' method with the filtered class' 'Frames', and use the save method eg. SaveToFile(). The 'AssignFramesByReference()' can be used on any level and obviously the original TStorageStream instance and the filter instance must be alive.
Note that, as the 'Frames' is originally part of the source instance any changes made to it are performed in the source instance actually - but if the source instance is not needed, eg. will not be saved, then this should not be a problem.
The sorting methods that have a 'Frame' in their name (eg. SortFramesByInteger()) are sorting the existing class. Methods without (eg. SortByInteger()) are returning a new list (a new 'TStorageStreamFramesBase' as a 'IStorageStreamFilter' interface) that is sorted, the orignal class is not modified.
If the Sort*() method's 'FrameName' is empty the frames are sorted by their current local root's (eg. string) value.
Grouping
Every frame has a 'GroupIdentifier' which is 0 by default. If 'GroupingIdentity' is True the 'GroupIdentifier' is stored when saving the stream.
TStorageStream has a 'GroupingContext' value that can be used to automaticaly work with the specified frames that belong to this group. Set 'GroupingContext' to a non-zero value and all newly added frames will have this 'GroupIdentifier'. All frame ID based look-up (getter and setter) functions will only return frames that belong to this group. On the contrary 'Index' based getter and setter function always work directly on frames specified by 'Index'.
'GroupingContext' is used in frame's context, to work with sub-frames, 'GroupingContext' needs to be set for the parent frame that has sub-frames that you wish to work with.
Versioning
Major version number is used internaly by the library, use the 'MinorVersion' to specify version of your stream before saving it if you wish.
When a new instance is created the 'MajorVersion' is automaticaly set to the latest implementation version. Saving the stream creates a stream with the latest version format. To save the stream with a legacy format set 'MajorVersion' to the desired legacy version number and set 'InLatestVersionFormat' parameter to 'False' when calling the save function.
When loading an existing stream 'MajorVersion' is set to the loaded stream format. Saving the stream creates a stream with the latest version format. To re-save the stream in the old version format set 'InLatestVersionFormat' to 'False' when calling the save function and the instance's 'MajorVersion' value format will be used when saving.
CRC
CRC32 error checking can be used on stream level and on frame level, both at the same time if needed.
Stream level: To calculate and write CRC32 for the stream set 'StorageStream.CRC.WriteCRC := True;'. When loading/opening a stream that has CRC32 specified in the stream, enable validation with 'StorageStream.CRC.CRCStatusScan := sscrcssValidate;' before loading or opening a stream. If this variable is set to 'sscrcssOmit' (default), then the CRC check will not be performed, the CRC32 value will be just parsed and not validated. Validating the CRC requires more processing so it's slower.
Frame level: To write CRC32 values for individual frames, enable the frame's 'CRCWrite := True'. Do this for all particular frames. To set this for all frames (and sub-frames) use SetWriteCRC32ForFrames(True), or to disable CRC32 writing 'SetWriteCRC32ForFrames(False)'. To check if a frame's CRC is valid use 'TStorageStreamFrame.CRC32Valid'. There is also a callback function, 'OnFrameCRC32Error' event for the base class, if that's assigned it will be called with the invalid CRC32 frame as the parameter on load/open if there is a CRC mismatch.
Turning on and using CRC functions (or assigning the 'OnFrameCRC32Error' event) slows down loading and saving functionality and is off by default.
Using the COM class
- Create an instance with (VBScript) 'set myobject = createobject("StorageStream.StorageStreamCOM")'
- Function with 'As' like 'GetAsString' work on the current frame. Functions without 'As' like 'GetStringByName' work with sub-frames of the current frame, 'ByName' postfix looks-up the first matching frame name automatically, 'ByIndex' postifixed functions expect a frame index explicitly.
- To get an interface for a frame use 'GetFrameByIndex()' or 'GetFrameByName()' functions. They give back an interface for the frame to access it's parameters or to add sub frames for example ('AddFrame()' which also gives back an interface for the newly created frame). This sub frame interfaces can be nested, to iterate over all the sub-frames or create multi-level sub-frame layouts.
- To limit loading of files that are generated by Your app. only, set the "load format filter" 4 ANSI character value with 'SetFormatFilter()' else all Storage Stream Files will be loaded/opened. The same 4 character value should be set before saving Your Storage Stream Files with 'SetFormatID()'. For example these 4 character values could be something like: 'MSSF' - My Storage Steram File eg. or any combination just use ANSI (non-unicode) characters.
- Functions that expect a 'LanguageID' require a 3 character ID like 'eng' for english.
- Functions that expect a 'Stream' require an IStream interface as the parameter.
- Hint: Use the included demo app. 'StorageStreamTutorialStructured.exe' to see the file's structure created by Your app..
Simple VBScript usage example:
option explicit
dim myobject, contentvalue, values
set myobject = createobject("StorageStream.StorageStreamCOM")
myobject.LoadFile "Storage Stream Test.3ssf"
wscript.Echo myobject.count
myobject.GetStringByIndex 0, contentvalue 'Get the first frame as a string
wscript.Echo contentvalue
set myobject=nothing
Delphi example for using virtual streams:
procedure TForm1.Button1Click(Sender: TObject);
var
SStream: IStorageStream;
Error: HResult;
FileStream: TFileStream;
StreamAdapter: TStreamAdapter;
NewFrame: IStorageStreamFrame;
begin
//* Acquire the class
SStream := CreateComObject(CLASS_StorageStreamCOM) as IStorageStream;
SStream.SetFormatID('MYDB');
//* Create a file stream
FileStream := TFileStream.Create('D:\Picture.jpg', fmOpenRead);
try
//* Create an IStream adapter fo the file stream
StreamAdapter := TStreamAdapter.Create(FileStream);
//* Add a new virtual frame with the IStream content
SStream.AddFrameVirtual('MyPIC', StreamAdapter, NewFrame);
NewFrame.Set_FrameType(ssftBinary);
//* Save the Storage Stream file, the IStream will be streamed to disk
Error := SStream.SaveToFile('D:\MyFile.ssf');
//* Display any errors, 0 means OK
Showmessage(IntToStr(Error));
finally
FreeAndNil(FileStream);
end;
end;
The COM methods that return frame are actually returning 'IStorageStreamFrame', the methods returning frame lists are actually returning 'IStorageStreamFrames' interfaces.
The COM methods that return and expect 'Stream' parameters are actually returning and expecting 'IStream' interfaces.
To get the interface out of the Variant in Delphi:
var
StorageStreamFrame: IStorageStreamFrame;
begin
StorageStreamFrame := IInterface(OutFrame) as IStorageStreamFrame;
end;
There is a little helper method in the base class 'SaveIStreamToFile()' to save the returned 'IStream' interfaces to file.
Encryption support
In 'StorageStream.pas' at the beginning of the unit the conditional define 'INDY_ENCRYPTION_SUPPORT' is on by default. It uses Indy's OpenSSL DLL support for DES encryption, so 'libeay32.dll', 'msvcr100.dll' and 'ssleay32.dll' are needed on the search path, eg. beside the EXE (according Win32, Win64 or OS specific DLLs, eg. 'libcrypto.so' and 'libssl.so' on Android (only 'armeabi-v7a' build included in the package, 'arm64-v8a' build needed for 64 bit https://github.com/IndySockets/OpenSSL-Binaries), also according to Indy version used, please see the Indy documentation for details.
Set the location of the DLLs with 'IdOpenSSLSetLibPath()' and then call 'IdSSLOpenSSLHeaders.Load' and don't forget to add to the uses list 'IdSSLOpenSSLHeaders' and the 2 .so files to deployment. On Windows just copying the DLLs beside the EXE will work automatically. Every call to the encryption functions that do not expect a crypting class will use this encrypting class.
Example code for Android:
IdOpenSSLSetLibPath(System.IOUtils.TPath.GetLibraryPath);
Showmessage(BoolToStr(IdSSLOpenSSLHeaders.Load, True));
If a message 'True' pops-up the 2 .so files were succesfully loaded.
If off, a default XOR encryption will be available only (TStorageStreamCryptXOR).
The default encrypting class is assigned to 'TStorageStream.Crypt'. To assign other class use 'TStorageStream.AssignCryptingClass()' - the object is automatically freed when freeing the 'TStorageStream' instance.
To use a custom class for another type of encryption please see 'IndyCryptDESClass.pas' or 'CryptXORClass.pas' for an example. The custom class needs to inherit from 'TStorageStreamCryptBase' and needs to have a unique GUID, also set 'EncryptionID' to '255'. The encryption GUID for the frames is only written if 'EncryptionID' is '255'. The 3 methods, as in the example, needs to be implemented. To use it create an instance and assign it with 'TStorageStream.AssignCryptingClass()'.
To save storage space there is a global flag 'GlobalEncryptionID' and a GUID value 'GlobalEncryptionIDGUID' that can be used to identify all encrypted frames, and the GUID value will not be stored for every frame, only as a single global value in the Storage Stream file.
It's possible to encrypt individual frames with different encryption methods then each frame will have it's own GUID stored in the file. For storing eg. 10 000 frames this means the there will be 10 000 GUIDs stored in the file, but it's possible to set a global value and if a particular frame uses a different method (and has a different GUID than the global value) then only the frame will have an own GUID stored in the file.
Use the encrypting class' .SetKey() method to set the encryption password, it must be all ANSI characters only (for the built-in DES and XOR class).
Use the frames' .Encrypt() and .Decrypt() functions to apply or remove the encryption, the 'Encrypted', 'EncryptionMethod' and 'EncryptionIDGUID' will be set automatically for the frame when calling these functions.
Example basic code:
var
StorageStream: TStorageStream;
begin
//* Create and save
StorageStream := TStorageStream.Create('SSCR');
try
StorageStream.Crypt.SetKey('password');
StorageStream.GlobalEncryptionID := True;
StorageStream.AddFrame('My test frame').SetText('Content 1').Encrypt;
StorageStream.AddFrame('My test frame 2').SetText('Content 2').Encrypt;
StorageStream.SaveToFile('D:\SS.dat');
finally
FreeAndNil(StorageStream);
end;
//* Load, decrypt and display
StorageStream := TStorageStream.Create('SSCR');
try
StorageStream.Crypt.SetKey('password');
StorageStream.LoadFromFile('D:\SS.dat');
StorageStream.FindFrame('My test frame 2').Decrypt;
Showmessage(StorageStream.FindFrame('My test frame 2').GetText);
finally
FreeAndNil(StorageStream);
end;
Note: when using the XOR class, if the frame data is compressed first (.Compress()), the encryption is a bit more secure. But note that for small data compression actually increases the data stream size. Also use long passwords (10 characters at least) when using XOR encryption.
Note: the COM class uses the Indy DES class by default so the needed DLLs mentioned above are required to use DES encryption with COM. To set encryption method use .SetEncryptionMethod(1) for XOR, .SetEncryptionMethod(2) for Indy DES.
Note: if bypassing the built-in encryption methods always set the frame's 'EncryptionMethod' value from '1' to '254', but note that it will not be possible to interpret the encryption scheme by 3rd parties (someone may use the same value in their own implementations).
Storing much data
When storing a lot, eg. 100 000 integer values it could be a better solution to create just 1 frame and add the values to it's 'Stream' (resizing to the needed size before).
A frame has '11' bytes for it's header + the frame name, a lot of space can be saved and it's quite simple to access particular values just by seeking to (integer) 4 * NeededIndex.
Useful information
|